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[Title of the Invention] 
10 DATA EDITING DEVICE AND DATA EDITING METHOD 

[Abstract] 

[Problem to be Solved] 

Enabling "association editing" of data recorded in 

15 the state of being associated with each other via 

additional information, without losing the meaning that 
the additional information originally has, such as the 
temporal meaning that time information has, other than 
the meaning of the additional information in terms of 

20 association. 
[Solution] 

When, as a result of recording the time when each 
of pen data and sound data was recorded is recorded as 
additional information, data associated with each other 
25 in terms of being simultaneously recorded via the 

additional information already exist, the association 
between pen data designated by a user from among pen 

- 1 - 
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data displayed on a screen, and sound data recorded at 
the same time as the designated pen data is canceled, 
and the pen data is newly associated with newly- 
recorded sound data. 
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[Claims for the Patent] 
[Claim 1] 

A data editing device for changing first data such 
as a character and second data such as a voice, the 
5 first data and the second data being associated with 
each other, so that the first data is newly associated 
with third data such as a voice, the third data being 
different from the second data, the device being 
characterized by comprising: 
10 means for giving an instruction for changing the 

association between the first data and the second data; 

means for selecting additional information for 
newly associating the first data and the third data 
with each other; 
15 means for designating the third data to be newly 

associated with the first data; and 

means for associating the first data and the third 
data with each other via the additional information. 
[Claim 2] 

20 A data editing device characterized by comprising, 

in relation to the first data and the third data 
associated with each other by the data editing device 
according to claim 1: 

means for searching for the first data; 

25 means for extracting the additional information at 

the time of the search by said first data searching 
means ; 
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means for searching for the third data associated 
via the additional information extracted by said 
extracting means; and 

means for replaying the third data searched for by 
5 said third data searching means. 
[Claim 3] 

A data editing method for changing first data such 
as a character and second data such as a voice, the 
first data and the second data being associated with 
10 each other, so that the first data is newly associated 
with third data such as a voice, the third data being 
different from the second data, the method being 
characterized by: 

obtaining an instruction for changing the 
15 association between the first data and the second data; 
selecting additional information for newly 
associating the first data and the third data with each 
other; 

designating the third data to be newly associated 
20 with the first data; and 

associating the first data and the third data with 
each other via the additional information. 
[Claim 4] 

A data editing method characterized by, 
25 in relation to the first data and the third data 

associated with each other by the data editing method 
according to claim 3: 
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extracting the additional information at the time 
of searching for the first data; 

searching for the third data associated via the 
additional information; and 
5 replaying the third data searched for. 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] 

10 The present invention relates to a data editing 

device and a data editing method that provide data with 
a plurality of additional information and store the 
data, and associate the data with each other via the 
additional information. 

15 [0002] 

[Conventional Art] 

Conventionally, various types of devices that 
record a trace of a pen drawn on a screen by hand 
(hereinafter referred to "pen data") and a sound 

20 (hereinafter referred to as "sound data") 

simultaneously using a mobile pen computer have been 
developed. For example, this type of device is 
provided with a contrivance to associate pen data and 
sound data with each other so as to retrieve and replay 

25 sound data "recorded simultaneously with" pen data 
designated by a user after recording the data. 
[0003] 

- 5 - 
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A method devised for this association is provided 
by simultaneously recording pen data and the times when 
the pen data were recorded, also simultaneously 
recording sound data and the times when the sound data 
5 were recorded, and specifying pen data and sound data 
recorded at the same time with reference to the times 
for the pen data and the sound data, respectively, as 
described in Japanese Patent Laid-Open No. 6-176171. 
[0004] 

10 An example described in this publication employs a 

method in which in order to specify "simultaneously- 
recorded data", a user first designates pen data 
displayed on a screen to acquire the time recorded 
simultaneously with the pen data, and searches for 

15 sound data including the time when the pen data was 
recorded, based on the recorded sound data times. 
[0005] 

[Problems to be Solved by the Invention] 

However, for devices that simultaneously record 

20 pen data and sound data, there is a demand for 

"association editing", such as associating pen data 
associated with existing sound data with newly-recorded 
sound data (replacing the previously-recorded sound 
data with new sound data) . In this case, the 

25 "association editing" can be performed by changing "the 
time (information) when the pen data was recorded" to 
the same time as the newly-recorded "time (information) 
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when the sound data was recorded" . 
[0006] 

However, when performing "association editing" 
with this method, "the time recorded simultaneously 
5 with the pen data" will be changed by the editing and 
thus the time when the pen data was recorded will be 
lost, resulting in the meaning that the information 
originally has, that is, when the pen data was recorded, 
being lost. In other words, it is impossible to 
10 designate a time and extract pen data in the time range, 
or automatically reproduce pen data in the order that 
they were recorded, for example. 
[0007] 

Therefore, an object of the present invention is 
15 to provide a data editing device and a data editing 
method that enable "association editing" of data 
recorded in the state of being associated with each 
other via additional information such as time 
(information), without losing the temporal meaning that 
20 the time (information) originally has. 
[0008] 

[Means for Solving the Problems] 

In order to achieve the above object, the present 
invention provides a data editing device and method for 
25 changing first data such as a character and second data 
such as a voice, the first data and the second data 
being associated with each other, so that the first 
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data is newly associated with third data such as a 
voice, the third data being different from the second 
data, the device and method being characterized by: 
obtaining an instruction for changing the association 
5 between the first data and the second data; selecting 
additional information for newly associating the first 
data and the third data with each other; designating 
the third data to be newly associated with the first 
data; and associating the first data and the third data 
10 with each other via the additional information. 
[0009] 

[Embodiments of the Invention] 
(Embodiment 1) 

Hereinafter, an embodiment of the present 

15 invention will be described with reference to the 

drawings. Here, the time (information) when each of 
pen data and sound data was recorded using a mobile 
terminal device is referred to as additional 
information, and the case where each of data 

20 simultaneously recorded (associated) via the additional 
information already exists will be described. 
Furthermore, a detailed description will be given 
particularly on a method for performing "association 
editing" in which the association between pen data 

25 designated by a user, from among pen data recorded on a 
screen by the user, and sound data recorded at the same 
time as the designated pen data is cancelled, and the 
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pen data is newly associated with newly-recorded sound 
data, without losing the meaning of the additional 
information (the time (information)) for the pen data. 
[0010] 

5 First, the appearance of the present mobile 

terminal device is shown in Figure 1 . The present 
device includes a pen 10 with which pen data is input, 
a tablet 11, a liquid-crystal panel 12 that displays 
each of pen data input by the pen 10, a microphone 13 
10 with which sound data is input, a speaker 14 that 

replays sound data, and various buttons. The pen 10 is 
provided for a user to directly enter characters or the 
like in the tablet 11. 
[0011] 

15 Figure 1 shows that: a "meeting material" is 

displayed on the liquid-crystal panel 12; and a 
recording start button 15 is operated to perform audio 
recording of a meeting via the microphone 13 while 
putting comments on the material via the pen 10. There 

20 already exists a device that displays a meeting 

material or the like on a liquid-crystal panel 12, and 
records characters or the like on this panel 12 using a 
pen 10. Also, the meeting material may be read out of 
a built-in disk, may also be received via a network, or 

25 may also be obtained by receiving one sent wirelessly. 
Figure 1 shows an example in which two pen data 1 and 2 
are input by a user at certain timings. One is a 
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letter "MEETING ROOM W (pen data 1) drawn to the 
right of "PLACE OF NEXT MEETING: NOT YET DECIDED" in 
the meeting material, which indicates the place of the 
next meeting that has been orally conveyed, and the 
5 other is a figure of a star sign (pen data 2) to the 
left of "- BRIEF HEADINGS" in the meeting material, 
which indicates importance. 
[0012] 

When each of the pen data 1 and 2 is recorded, 

10 respective sound data obtained via the microphone 13 
are associated with them. When pen data 1 was drawn, 
the chairperson of the meeting said "we will have the 
next meeting at the meeting room in block A", and thus, 
this sound data (sound data 1) is associated with the 

15 pen data 1. Also, when the pen data 2 was drawn, the 
voice, "this is important", was given by the user 
drawing the pen data, and is associated with the pen 
data 2 as the sound data (sound data 2) . Although 
loading of sound data is started by operating the 

20 recording start button 14, only when sounds with a 

volume equal to or exceeding a predetermined volume are 
detected, such sounds are loaded (stored) as sound data. 
For this mechanism, already-established techniques can 
be employed. However, not all loaded sound data are 

25 associated with pen data, and sound data at a 

substantially same time as the time when pen data is 
recorded (including a time near that time or the case 
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where these times overlap) is associated with the pen 
data. "Time(s)" here mentioned may be absolute time(s) 
(clock time[s]) or may also be time(s) in the mobile 
terminal device. 
5 [0013] 

Figure 4 schematically shows a data structure in 
which pen data and sound data are recorded in the state 
of being associated with each other via additional 
information. For the pen data 1 and 2, times are 

10 stored in an additional information item of the type 
"Time". Meanwhile, also for the sound data 1 and 2, 
times are recorded in an additional information item of 
the type "Time". In the case of Figure 4, the times 
when the pen data were recorded and the times when the 

15 sound data were recorded are the same, the pen data are 
associated with the respective sound data. 
[0014] 

The pen data 1 is drawn during the time "'10"00- 
'10"12" (from 10:00 to 10:12), and the sound data 1 is 

20 recorded during the same time " ' 10"00- ' 10"12" (from 
10:00 to 10:12). In this case, the times (sections) 
entirely overlap, and thus the pen data 1 is associated 
with the sound data 1 via additional information of the 
type, time. The same applies to the pen data 2 and the 

25 sound data 2. 
[0015] 

Figure 2 is a diagram showing the overall 
- 11 - 
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configuration of the present device. A method for 
performing association editing of data associated with 
each other via additional information (time 
information) shown in Figure 4 will be described using 
5 Figures 2 and the flowchart shown in Figure 3. Figure 
3 shows a procedure for performing association editing. 
The pen data 1 and 2 shown in Figure 4 are stored in 
advance in a storage unit 20 via the aforementioned 
procedure, and displayed on the tablet 11 (the screen 
10 of the liquid-crystal panel 12) by means of a display 
control unit 21. The sound data shown in Figure 4 is 
also stored in a storage unit 30, and can be replayed 
from a speaker 13 by means of a replay control unit 29. 
[0016] 

15 Here, an operation procedure to perform 

"association editing" in which the association between 
the pen data 2 displayed on the screen and the sound 
data 2 recorded at the same time as the pen data 2 is 
cancelled and the pen data 2 and newly-recorded sound 

20 data (sound data 3) are newly associated will be 
described . 
[0017] 

First, a user views pen data displayed on the 
screen, and designates the pen data 2 from among them. 
25 For this pen data designation, a mode for performing 
"association editing" is designated (via a button or 
the like not shown: S40) to activate a pen data 
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designation unit 22 shown in Figure 2. The user views 
the pen data displayed on the screen to find the pen 
data 2, and then touches the part of the liquid-crystal 
panel 12 where the pen data 2 is displayed, via the pen 
5 10. As a result, the pen data designation unit 22 

detects where on the tablet 11 is touched via the pen 
10 in terms of coordinates (Xt, Yt) on the tablet 11. 
The pen data designation unit 22 calculates the 
distance between each of the coordinates (Xn, Yn) of 

10 all the pen data stored in the storage unit 20, and 

these coordinates (Xt, Yt) according to, for example, 
the formula ((Xt-Xn)**2 + (Yt-Yn)**2) to search for pen 
data including coordinates with the smallest 
calculation result value. 

15 [0018] 

In this example, only two pen data, i.e., the pen 
data 1 and the pen data 2, are provided as the pen data, 
and since the user has touched on the pen data 2, the 
pen data designation unit 22 designates the pen data 2. 
20 [0019] 

Next, the user designates a mode for performing 
association editing in which the pen data 2 and sound 
data 3 to be newly recorded are associated with to each 
other, (via a button or the like not shown: S41). As a 
25 result of this designation, an editing instruction unit 
23 in Figure 2 is activated. Upon the user pressing a 
new sound data association button 16 after selecting 
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the pen data 2, the editing instruction unit 23 detects 
which pen data is designated by the pen data 
designation unit 22, and gives an instruction to 
associate the pen data 2 and sound data 3 to be newly 
5 recorded with each other. Then, the user presses the 
recording start button 14 to start the recording of 
sound data 3, input a voice, "this is the point" via 
the microphone 17, and presses a recording stop button 
15 to stop the recording, and consequently the sound 

10 data 3 is stored in the storage unit 30. At this time, 
also, the sound data 3 is stored in the storage unit 30 
with the time when it was stored added to additional 
information of the type "Time". This state is shown in 
Figure 5 . 

15 [0020] 

Next, the additional information type used for 
editing is selected via an editing additional 
information selection unit 24 in Figure 2 (S42) . With 
reference to the additional information for the pen 

20 data 2 and the sound data 3 that had been associated 

with the pen data 2, the editing additional information 
selection unit 24 detects that only additional 
information of the type "Time" is provided. An 
additional information selection table storage unit 25 

25 in Figure 2, as shown in Figure 7, stores priority 
information relating to which type of additional 
information is prioritized to perform association, in 
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an additional information selection table, and based on 
this, selects which additional information is used to 
associate the pen data 2 and the sound data 3. 
[0021] 

5 The additional information selection table shown 

in Figure 7 shows that, for example, at the time of 
association editing, association was already performed 
in the past using the additional information of the 
type "Time", additional information of the type "Edit" 

10 in the next priority is used. Upon additional 

information used for association editing being selected 
by an editing additional information selection unit 26, 
an association change unit 27 in Figure 2 releases the 
sound data 2 already associated with the pen data 2, 

15 and newly associates the pen data 2 with the sound data 
3 (S44 and S45) . 
[0022] 

First, the association change unit 27 searches for 
all the pen data and sound data having additional 

20 information of the type selected by the editing 

additional information selection unit 26 to figure out 
the integer greater than the maximum value of them by 
one. Consequently, this value is the one and only one 
as far as the selected additional information is 

25 concerned (S43) . Next, using this value, the 

association change unit 27, as shown in Figure 6, 
provides the additional information of the type "Edit" 
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using an ID of "3" (S44 and S45) . 
[0023] 

A method for replaying data on which association 
editing has been performed as described above will be 
5 described using Figure 2. First, as in editing, pen 
data displayed on the screen is designated by means of 
the pen data designation unit 22 in Figure 2. Then, 
the reference additional information selection unit 24 
finds out which type of additional information is used 
10 to perform the association, with reference to the 

content in the additional information selection table 

storage unit 25. 

[0024] 

In the example in Figure 6, the pen data 2 is 
15 provided with two types of additional information, i.e., 
the type "Time" and the type "Edit", but in the 
additional information selection table in Figure 7, the 
"Edit" type is prioritized over the "Time" type, and 
thus, the reference additional information selection 
20 unit 24 detects that the association has been made 

using the "Edit" type. Next, the reference additional 
information selection unit 24 acquires the information 
that the "Edit" type is provided with an ID of "3", 
with reference to the storage unit 20. Then, a sound 
25 data search unit 28 specifies the sound data with its 
"Edit" type provided with an ID of "3" from the sound 
data stored in the storage unit 30, by referring to the 
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values of the "Edit" type for all the sound data. This 
ID is the one and only one in association editing, and 
accordingly, a plurality of sound data will not be 
found. The sound data 3 with its "Edit" type provided 
5 with an ID of "3" is specified as described above. 

Then, the replay control unit 29 reads the voice data 
of the sound data 3 specified by the sound data search 
unit 28 out of the storage unit 30 and replays it from 
the speaker 13 equipped with the housing. 
10 [0025] 

As described above, in association editing, the 
pen data 2 and the sound data 3 are associated with 
each other using additional information prioritized 
over the additional information used for the 

15 association between the pen data 2 and the sound data 2 
that had been associated with the pen data 2 before the 
editing, and subsequent association is performed using 
additional information with the highest priority, and 
accordingly, the association according to the 

20 additional information provided before becomes invalid. 
[0026] 

As a result of providing the above-described 
mechanism, it is possible to conduct a search using 
time information provided for association before 
25 editing, for example, finding out the trace of the pen 
10 drawn by hand which was recorded during the time 
( ' 10"00- ' 10"12) , in terms of another meaning, i.e., the 
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temporal relationship, that the time information has 

even after the editing. 

[0027] 

Also, as another example indicating the 
5 effectiveness of the present invention, sound data 
associated with the handwritten traces of the pen 10 
can automatically be replayed sequentially in the order 
that the traces were recorded. In the editing 
according to the present invention, the time 

10 information provided with the handwritten traces of the 
pen 10 will not be changed, and thus, in automatic 
replay, the sound data can be replayed in the same 
order as the order of the handwritten traces of the pen 
10 in replay before the editing. In addition, for 

15 sound data for a handwritten trace of the pen 10 on 

which editing has been performed, newly-recorded sound 

data is replayed. 

[0028] 

(Embodiment 2) 

20 While in embodiment 1, the reference additional 

information selection unit 24 performs selection based 
on the content in the additional information selection 
table storage unit 25, "association tags" may be 
provided to the storage unit 20 and the storage unit 30 

25 at the time of association editing as shown in Figure 8 
to store which additional information is used for 
association, and additional information may be selected 
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with reference to them. 
[0029] 

(Embodiment 3) 

While in embodiment 1, additional information of 
5 the type "Edit" has been used as additional information 
used for association in editing, for example, where 
editing target sound data is an independent file, the 
file name, etc., may be used. 
[0030] 

10 [Advantage of the Invention] 

The present invention enables association editing 
to be performed without losing the meaning that the 
previously-provided additional information has. 
[Brief Description of the Drawings] 
15 [Figure 1] 

Figure 1 is an appearance of the present invention. 
[Figure 2] 

Figure 2 is a diagram showing an overall 
configuration of the present invention. 
20 [Figure 3] 

Figure 3 shows a procedure for an association 
change . 
[Figure 4] 

Figure 4 shows a state of data before association 
25 editing. 

[Figure 5] 

Figure 5 shows a state of data during association 
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editing . 
[Figure 6] 

Figure 6 shows a state of data after association 
editing . 
5 [Figure 7] 

Figure 7 shows the content of an additional 
information selection table. 
[Figure 8] 

Figure 8 shows data provided with association 
10 additional information tags. 
[Description of Symbols] 

20 storage unit 

21 display control unit 

22 pen data designation unit 
15 23 editing instruction unit 

24 reference information selection unit 

25 additional information selection table storage 
unit 

26 editing additional information selection unit 
20 27 association change unit 

28 sound data search unit 

29 replay control unit 

30 storage unit 



Japanese Patent Laid-Open No. 11-65752 



Figure 1 

1 PEN DATA 

2 PEN DATA 
5 10 PEN 

11 TABLET 

12 LIQUID-CRYSTAL PANEL 

13 SPEAKER 

14 RECORDING START BUTTON 
10 15 RECORDING STOP BUTTON 

16 NEW SOUND DATA ASSOCIATION BUTTON 

17 MICROPHONE 

#1 TOPICS TO BE INCLUDED IN MAGAZINE 

#2 ENTERTAINMENT 

15 #3 WEATHER FORECAST 

#4 REQUESTS 

#5 HEAVY USE OF PHOTOS 

#6 BRIEF HEADINGS 

#7 PLACE OF NEXT MEETING: NOT YET DECIDED 

20 

Figure 2 

21 DISPLAY CONTROL UNIT 

22 PEN DATA DESIGNATION UNIT 

23 EDITING INSTRUCTION UNIT 

25 24 REFERENCE ADDITIONAL INFORMATION SELECTION UNIT 

2 5 ADDITIONAL INFORMATION SELECTION TABLE STORAGE 
UNIT 
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2 6 EDITING ADDITIONAL INFORMATION SELECTION UNIT 

2 7 ASSOCIATION CHANGE UNIT 

2 8 SOUND DATA SEARCH UNIT 

2 9 REPLAY CONTROL UNIT 

5 #1 PEN DATA 

#2 ADDITIONAL INFORMATION 

#3 MEETING ROOM A 

#4 THIS IS IMPORTANT 

#5 THIS IS POINT 

10 #6 SOUND DATA 



Figure 3 

S4 0 DESIGNATION OF PEN DATA 

S41 INSTRUCTION TO PERFORM ASSOCIATION EDITING 
15 S42 SELECTION OF ADDITIONAL INFORMATION TYPE USED FOR 
EDITING 

S43 ACQUISITION OF ONE AND ONLY VALUE (ID) IN TERMS OF 
SELECTED ADDITIONAL INFORMATION 
S4 4 PROVIDE PEN DATA WITH ID AND STORE IT 
20 S45 PROVIDE SOUND DATA WITH ID AND STORE IT 
#1 START 
#2 END 



Figure 4 
2 5 #1 PEN DATA 

#2 ADDITIONAL INFORMATION 
#3 VOICE DATA 
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#4 MEETING ROOM A 
#5 THIS IS IMPORTANT 

Figure 5 

5 #1 PEN DATA 

#2 ADDITIONAL INFORMATION 

#3 VOICE DATA 

#4 MEETING ROOM A 

#5 THIS IS IMPORTANT 

10 #6 THIS IS POINT 

Figure 6 

#1 PEN DATA 

#2 ADDITIONAL INFORMATION 

15 #3 VOICE DATA 

#4 MEETING ROOM A 

#5 THIS IS IMPORTANT 

#6 THIS IS POINT 

20 Figure 7 

#1 PRIORITY 

#2 ADDITIONAL INFORMATION TYPE 

Figure 8 
2 5 #1 PEN DATA 

#2 ADDITIONAL INFORMATION 
#3 VOICE DATA 
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#4 MEETING ROOM A 

#5 THIS IS IMPORTANT 

#6 THIS IS POINT 

#7 ASSOCIATION TAG 
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